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Abstract. We study a simple extended model of oscillator neural networks capable 
of storing sparsely coded phase patterns, in which information is encoded both in the 
mean firing rate and in the timing of spikes. Applying the methods of statistical 
neurodynamics to our model, we theoretically investigate the model's associative 
memory capability by evaluating its maximum storage capacities and deriving its basins 
of attraction. It is shown that, as in the Hopfield model, the storage capacity diverges as 
the activity level decreases. We consider various practically and theoretically important 
cases. For example, it is revealed that a dynamically adjusted threshold mechanism 
enhances the retrieval ability of the associative memory. It is also found that, under 
suitable conditions, the network can recall patterns even in the case that patterns 
with different activity levels are stored at the same time. In addition, we examine the 
robustness with respect to damage of the synaptic connections. The validity of these 
theoretical results is confirmed by reasonable agreement with numerical simulations. 



1. Introduction 

One of the important but unsolved problems in neuroscience is to determine how 
information is coded in neuronal activities. Most of the traditional neural network 
models consisting of binary units are constructed on the assumption that information 
is coded only in the mean firing rate of the neurons. Although these models have 
provided us with theoretically interesting information, they ignore many dynamical 
aspects of neuronal activities. In fact, oscillatory activity appears to be ubiquitous in 
many neuronal systems. For example, some recent biological experiments have revealed 
that spatially synchronous oscillations of neuronal ensembles are dependent on the global 
properties of the external stimulus. It has been suggested that such synchronization is 
computationally significant in information processing The hippocampus is also one 
of the areas in which neuronal synchronization is observed and is believed to play an 
important role in memory processing. 

With these new findings, many models concerning the dynamical aspects of 
neurons and memory processing have been proposed [§, H, ^. Among these, models 
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consisting of networks of oscillator components are particularly attractive, owing to 
their mathematical tractability. Like the Hopfield model, the simplicity of these models 
allows us to obtain useful analytic results. In fact, it has been reported that under 
suitable conditions, the oscillator network models model associative memory, and we 
can theoretically evaluate their maximum storage capacities and basins of attraction 



However, such phase oscillator models are based on the unrealistic situation that all 
the components in the network are always in the firing state. In fact, real neurons can be 
in either the firing or non-firing state. In addition, it is known that the level of activity 
in our brain is very low. (i.e., at any given time only a small percentage of neurons are in 
the firing state.) This situation is termed sparse coding. From the results of theoretical 
studies of associative memory with binary units, it has been found that the storage 
capacity diverges as — 1/alna as the firing rate a becomes small |16|, [18|, |19|, [20[] . 

It thus seems that to faithfully to capture the essential dynamics of real oscillatory 
neuronal systems, it is necessary to extend the oscillator model to treat the non-firing 
state as well as the firing state. In this paper, we present a simple extended oscillator 
neural network model of this nature. Using this extended model, we study its associative 
memory ability to recall sparsely coded phase patterns in which some neurons are in 
the non-firing state and others encode information in the timings of spikes. 

In the next section, we first review the theoretical basis of the phase oscillator model 
and propose an extend version of the oscillator model to treat non-firing states. In the 
analysis of this model, we estimate the maximum storage capacity, derive the basin of 
attraction, and evaluate the quality of recalled memories. Then, we find that when we 
define the threshold as a dynamical variable in a certain manner, the size of the basin 
of attraction can be increased. Embedding patterns with different activity levels and 
the influence of synaptic dilution are also studied. 

2. Theories of Oscillator Neural Networks 

2.1. Phase Oscillator Model 

Let us first start with a survey of the theoretical basis of the phase oscillator model. We 
assume that neurons fire periodically and interact weakly with each other. In general, 
although such a neural system can be described in terms of many internal dynamical 
variables, it is well known that it can be reduced to a system of simple coupled phase 
oscillators 0]. In this form, we can characterize the state of the i-th neuron by a single 
variable, (f)i{t), which is referred to as the phase. This variable represents the timing of 
the neuronal spikes at time t. A typical reduced equation takes the form 



where Jij and characterize the interaction between the i-th and j-th neurons. 
Assuming that all natural frequencies uji{t) are equal to some fixed value Uq, we can 
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eliminate the ujiit) term in Eq.(|l|) by redefining through (piit) +ujQt. When 

we represent the state of the i-th neuron by the complex form Wi{t) = exp(z0j(t)), 
Eq.(|l]) can be written in the alternative form 

= Im) - Ht)wm), h,{t) = j:Q,w,{t), (2) 

where the complex variable Cij = Jij exp{if3ij) represents the effect of the interaction 
between the i-th and j-th neurons and hi(t) is the complex conjugate of hi(t). 
Considering the fact that all neurons relax toward the equilibrium state satisfying the 
relation Wi = hi/\hi\, we can simplify the above to the following discrete time system: 

h(t) ^ 

m+i) = jj^y h.it) = j:a,w,it). (3) 

This is known as a phase oscillator network model. It can be thought of as a synchronous 
updated version of the phase oscillator neural network. 

2.2. Extended Model 

The weakness of the model described by Eq.(|^) is that it can only be used to treat the 
firing state. We wish to extend this model so that it has the ability of retrieving sparsely 
coded phase patterns. In Eq.(^, hi(t) can be regarded as the local field produced by all 
other neurons. This field determines the state of the i-th neuron at the next time step. 
In a real neuron, when the membrane voltage is less than some threshold, the neuron 
generates no neuronal spikes. Therefore, it is reasonable to extend the above model so 
that the generation of spikes depends on the strength of the local field. Based on this 
consideration, for oscillator neural networks we propose the following generalized model: 

hit) ^ 

w,{t+i) = fmmjjrFw-. h^it) = Y.c^,w,{t). (4) 

I *V yl 

In this paper, we assume that /(x) = Q{x — H), where 6(x) is a step function defined 
by 

{1 for X > 
for X < 0, 

and H is a. threshold parameter controlling the activity of the network. 

Figure[l|(a) displays the function 9(x — if), and (b) illustrates the dynamical change 
at updating. Since the amplitude depends on the threshold H, it has a strong 

influence on the activity of the system. 

Now, we define a set of complex patterns denoted by = Af exp(i0f ) (/i = 1, ■ ■ ■ , P 
; i = 1, ■ ■ ■ , A^), where P is the total number of patterns and is the total number 
of neurons. The variables 6^ and represent the phase and the amplitude of the 
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i-th neuron in the /i-th pattern, respectively. For theoretical simplicity, we choose 
independently with the probability distribution: 

{1 for firiner state with probability a , , 

(6) 
for silent state with probability 1 — a. 

For the firing state, is chosen at random from a uniform distribution between and 
27r. Note that all the patterns have the same mean firing rate, a. As the learning rule, 
we adopt a generalized Hebb rule taking the form 

c« = j^|:fr§. (7) 

This relation is based on information obtained from experiments on biological systems 
that when the i-th and j-th neurons are firing simultaneously, the connection between 
them is enhanced, and otherwise no modification occurs. 

2.3. Theoretical Analysis 

To analyze the recalling process theoretically, we must introduce several macroscopic 
order parameters. The load parameter a is defined by a = P/N. The overlap M^{t) 
between Wi{t) and at time t is defined by 

1 ^ ~ 

M^{t)=m^{t)eM^V'm = ^Ee^W^.W- (8) 

In practice, owing to rotational symmetry, the similarity between the state of the system 
Wiit) and the /i-th pattern can be measured by m^(t), the amplitude of M^(t). Now, 
let us consider the situation in which the system is retrieving the pattern Q, that is, 

m(t) =m^(t) -0(1), m^{t) ^o(^] (fi^l). (9) 



The local field hi{t) can be separated as 

h,{t) = M\t)il + z,{t) = |CV(t)e^(^'W+''') + ^^^t), (10) 



where Zi{t) is defined by 

1 



= E iUW^it)- (11) 



The first term in Eq.(|TO|) is the signal driving the system to recall the pattern, and the 
second term can be regarded as the noise arising from the other learned patterns. It is 
the essence of our analysis that we treat the second noise term as a complex Gaussian 
noise characterized by 

< z,{t) >i= 0, < \zi{t)\^ >i= 2a\t). (12) 

Under the above assumptions, we study the properties of this network by applying the 



methods of statistical neurodynamics ||2T|, ^ ^ To begin with, we calculate the 
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overlap at time t + 1. From Eqs.(|]), and (0), M^(t + 1) is given by 

1 ^ ~ h (t) 

M\t + 1) = m{t + 1) expi^^\t + 1)) = ^ E ^1/(1/^. Wl)^ 



Now, we assume that the phase of M^(t + 1) is almost constant, that is, (f^{t + 1) ifQ. 
The validity of this assumption is supported by the results of preliminary numerical 
simulations. Owing to the rotational symmetry of the complex Gaussian noise, we can 
replace Zj(t) with Zj{t) exp[i{ifo + After some calculations, in the limit N — > cxd, 
we obtain 

K'^i)-^£i^ji/(ii^jH')^^,wi) |j||:;:;;g:;i 

where <^ ^^(i) represents the average over the complex Gaussian noise z{t) defined 
by Eg . (|T2|) . To calculate Eq.(|T3l) numerically, we need the value of which is the 
variance of z{t). Thus, in the next step, we consider the relation between Zi{t + 1) and 
Zi{t), from which we can obtain the relation between a{t + 1) and cr{t). From Eq.(|ll]), 
the noise at time t + 1 can be written as 

-^i:gg/(i^rw+''?wi) |gg;g:;| . (14) 

where h~'^{t) and h'^it) are defined by 

hnt) = ^ E ^fe.w.it), h){t) = j^Y^m^^ity (15) 

k,fj,y^u k 

Note that these functions satisfy the relation hj{t) = hj'^{t) + h'^{t). We can carry out 
the summation in Eq.(p!4D under the assumption that h~'^{t) is independent of . Doing 
so, we obtain 

Zi{t + l) ^ K{m{t),a{t)) + z,{t)G{m{t),a{t)) (16) 
K{m{t),a{t)) = ^ E ^'^^ViMm^^ (17) 



. 2 2\zit)\ , 



z(t) 
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(19) 



Note that Eqs.(|T6|), (|T^ and (0) are needed to calculate a in both the equilibrium and 
the non-equilibrium cases we will describe in the next two sections. 

2.4- Equilibrium State 

In this section, we consider the equilibrium state of our model, in which Zi{t) is constant. 
Applying Zi(t + 1) = Zi(t) = zi to Eq.(|^), we obtain 
K 

Using this equation, we immediately obtain 

2a' =< \z,\' >,= ^Y?^, (20) 

where 

\K\' = a{{af{\m + z\) + {l-a)f{\z\)))^ (21) 
= aQ. (22) 

Consequently, we find that the equilibrium state satisfies the equations 

^= ((fi\^ + ^\)^^^)) (23) 



\m + z\ 




(25, 

^ ^ \ 2 2\z\ 
Q = {{af{\m + z\) + {l-a)f{\z\)))^. (26) 

Solving the above equilibrium equations, we can find two types of solutions. If the 
load parameter a is smaller than a certain value ac, there exists a solution for which 
the overlap m between the system and the pattern is not zero. As m 7^ implies that 
the system has a correlation with the retrieved pattern, this solution corresponds to 
the retrieval state. Then, if a is larger than ac, there is only one solution, and for this 
solution m = 0. This solution therefore corresponds to the non-retrieval state. 

The critical load parameter ac is called "the maximum storage capacity". We 
now examine our theoretical results by comparing them with results from numerical 
simulations. In Fig.0(a), we show the dependence of ac on several parameters, such as 
the threshold H = 0.3,0.5,0.8 and the activity level a. In (a). We can see that as the 
activity level a decreases, the storage capacity ac increases for each H. In particular, 
in the limit a ^ 0, we numerically find that the storage capacity diverges in proportion 
to — 1/alna, as in the case of the Hopfield model. The maximum storage capacity as a 
function of a and H is illustrated in (b). It is shown there that for any H, the storage 
capacity diverges as a decreases to zero. 
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2.5. Non-equilibrium State 

In this section, we derive dynamical equations to study the retrieving process. To obtain 
a recursion equation for a, we start with Eq.(|l^). Squaring Eq. ([T6|) , we obtain 

2a^{t + 1) = aQ{m{t), a{t)) + 2a^{t)G^{m{t), a{t)) 

+ 2Re {{K{m{t), a(t))5,(t)G(m(t), , (27) 

where K{m{t),a{t)) and G{m(t),a{t)) are given by Eqs.(p!7D and (p!8|), and 

Q{m{t), a{t)) = {{ af{\m{t) + z{t)\) + (1 - a)f{\z{t)\) . (28) 

To proceed with the calculation, we must estimate the time correlation of the noise. 
For the first-order approximation, we ignore the temporal correlation of Zi{t). We then 
obtain 

a^t + 1) = |g(m(t), a{t)) + a\t)G\m{t), a{t)) 

+ aa^m{t + l)m{t)G{m{t),a{t)). (29) 

This corresponds to the Amari-Maginu theory in the case of traditional neural networks. 

For the second-order approximation, we take into account the fact that Zi{t) is 
correlated only with Zi{t — 1), while all correlations with Zi{t') ioT t' < t—1 are ignored. 
In this case, a(t + 1) can be obtained from 

a^t + 1) = |Q(m(t), ait)) + a\t)G\m{t), ait)) 

+ aG{m{t),a{t))X{t + l,t) (30) 
+ aa^m{t + l)m{t - l)G(m(t), a{t))G{m{t - 1), a{t - 1)), 

where Q{m(t),a(t)) is given by Eq. (pHD , G{m(t),a{t)) is given by Eq.([T8|), and 

X{t + l,t) = Re< Wj{t + l)Wj{t) >j . (31) 

Here, 



is needed to evaluate Eq . (|3lD . Note that p{t, t — 1) is the correlation coefficient between 
z(t) and z{t — 1). For the second order approximation, consequently, the retrieving 
process of the network is described by the equations 



aiV^'^^'^^"^^' ^''^'\\^}\m{t) + zM 



a^{t +1) = |g(m(t), ait)) + a^it)G^im{t), a{t)) 

+ aG{m{t),a{t))X{t+l,t) (34) 
+ aa^m(t + l)m{t - l)G(m(t), a{t))G{m{t - 1), a{t - 1)) 

X{t + 1, t) = Re< Wj{t + l)Wj{t) >j (35) 
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(38) 



(37) 



(36) 



For initial conditions, we choose o"^(0) = aa/2, X{0,—1) = and X{1,0) = 
a^m(l)m(0). We choose many values of the initial overlap m(0) and carry out numerical 
calculations for each. In this way, we determine the lower bound, under which the 
network fails to retrieve the pattern. 

For higher-order approximations, we could derive similar generalized equations 
describing the retrieval dynamical process. However, as shown in FigJ^, the theoretical 
prediction at second order gives reasonable agreement with the numerical simulation 
in contrast to that at first order. We thus conclude that it is sufficient to use the 
second-order approximation for the theoretical analysis in this paper. 

Figure]^ displays a phase diagram for m(0)c and m{oo) at various mean activity 
levels a and thresholds H. The solid lines and the data points here correspond to the 
theoretical and the numerical results, respectively. It is seen that even in the region 
satisfying a < ac, the system cannot retrieve the pattern if the initial overlap m(0) is 
smaller than m(0)c. The boundary corresponding to m(0)c is represented by the lower 
curves in Fig.§. Thus, for m(0) > m(0)c, m(t) reaches the value of the upper curves 
m(oo), while for m(0) < m(0)c, m{t) decreases to zero. We should note that the basins 
of attraction remain wide even near ac- This can be thought of as an advantage of 
associative memory. 

To predict the correct behavior of the retrieval dynamics, it is necessary for the 
time correlation of the noise terms to be taken into account, and thus the second-order 
approximation discussed above is necessary. In the case we consider, this order is also 
sufficient, but interestingly, it has been reported that the fourth-order approximation 
is necessary when a = 1 and H = The question arises why the second-order 

approximation is not sufficient in this case, while it is sufficient in the case we consider. 
The key point here is that the order of the last term in Eq.(|3^) is proportional to the 
square of the activity level a^. For a < 1, this factor weakens the influence of the 
time correlation on the recalling process. Consequently, for a sparse coded pattern with 
a < 1, even the second-order approximation results in reasonable agreement with the 
numerical results. 

2.6. Sequence Generator 

In this section, we consider the case in which the network retrieves a cyclic sequence of 
P patterns associatively, say —> ^■'^ ■. In order to allow for 
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such a process, we employ synaptic connections of the form 

c. = ^|:«r't. (39) 



In a manner similar to that in the derivation of Eqs.(|29D and (0), we obtain the following 
equations: 



a^(t + 1) = |Q(m(i),<T(«)) + <T=(i)G'(m(i).<T(«)) (40) 

Note that, since the target pattern changes from time to time, for the definition of the 
overlap we adopt m(t) = m^{t) = \-^J2jLi^jWj{t)\, where fx is the number of the 
target pattern at time t. Here G{m(t),a(t)) and Q{m{t),a{t)) are the same as those 
defined by Eqs. (pTj) and ([38|), respectively. We should note that in the limit N oo, 
the last term in Eqs. (p9|) and (p^ ) vanishes in Eq. (|iO|) , because the effect of the time 
correlation can be ignored in the above derivation. Therefore, in a sequence generator, 
it is expected that our theoretical prediction is almost exact. 

Figure|^(a) displays a phase diagram obtained from our theoretical analysis. All 
the lines of theoretical results agree with the numerical ones quite well, as expected. 
This agreement suggests that higher-order approximations will result in even better 
agreement and leads us to believe that our theoretical derivation is valid. In (b), it is 
also shown that, like auto-associative memory, the storage capacity diverges in the limit 
a — * 0. This is regarded as expressing the meaning that in the limit of sparse coding 
(i.e., as a ^ 0), we can embed an infinite length of sequential patterns. 



2. 7. Dynamically Adjusted Threshold 

From Fig.^ it appears that the basin of attraction for our model is smaller than those 
in the binary model and the phase oscillator. In this section, we attempt to increase 
the size of the basin of attraction by defining the threshold as a dynamical variable that 
is proportional to the standard deviation of the noise. Therefore, assuming the same 
condition as in Sec j2.2| , we add the equation 

H{t) = V-2\naa{t), (42) 

where H{t) is the threshold at time t. This choice for the form of H{t) is made to 
insure the relations m ~ 1 and [firing rate] ~ a, which cause the Hamming distance 
between the state of the neuron and the retrieved pattern to be small Comparing 
Fig.p with Fig.^, it is seen that the basin of attraction can be enlarged by introducing 
the dynamically adjusted threshold into our model. Therefore, when we introduce the 
dynamically adjusted threshold, our model has an advantage similar to that of the 
binary model and the phase oscillator. 
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3. Patterns Generated with Different Firing Rates 

To this point, we have assumed that the activity levels are equal for all patterns. 
However, it is likely that the actual activity level depends on the pattern which 
the network is retrieving presently, in other words, on the content of the required 
information. Unfortunately, using a traditional neural network, we encounter difficulties 
in storing patterns with different activity levels simultaneously. In this section, we 
demonstrate that, unlike traditional models, the proposed oscillator model can easily 
store multiple patterns with different activity levels using a simple Hebbian learning 
rule. 

Let us consider a set of complex patterns defined by 

f fli for 1< u < Pi 
Probm\ = l] = { „ -/-^ ' p (43) 

I 02 for _ri + 1 < /i < P, 

where generally ai 7^ 02. Thus, the total number of the patterns with activity level oi is 
Pi, while the total number of patterns with the activity level 02 is P2 = P — Pi. Using 
the above patterns, grouped into two different activity levels, we examine whether the 
network can retrieve patterns having different firing rates. To retrieve such patterns, we 
use the following modified form of the learning rule Eq.(^: 

0.-^Zm^^Zjir (44) 

Note that this is slightly different from the covariance rule adopted in the context of 
the learning of sparsely coded patterns. Owing to the rotational symmetry of the phase 
distribution in the patterns, Cij can be defined in terms of the patterns themselves, 
rather than the difference between the patterns and the average activity. 

With the method described in Sec.|^, we can derive both equilibrium and dynamical 
equations. The results have the same forms as those in Sec.|^. However, here the value 
ai should be used in place of the activity parameter a in the previous equations, because 
the retrieval pattern C,i has an activity level ai. Let us define the load parameter as 
ai = Pi/N with respect to the retrieval pattern. (The usual load parameter is defined as 
a = P/N = (Pi + P2)/N.) The theoretical analysis yields aic = const. — a^, where a\c 
is the maximum storage capacity for oi, and 02 is the storage capacity for 02. Figure^ 
displays a\c as a function of 02 = P2/N . We can see from Fig.|^ that aic = a{ai) — 02, 
where the constant value a(ai) is given by the equations in (^), (p^), (|25|) and (p6D for 
a = ai. For the maximum storage capacity associated with the activity level a2, we 
have = 0(02) — ai. 

Let us consider the case Pi = P2. Note that in this case, ai = 02 and the total 
storage capacity a has the relation as a = 2ai = 2a2- We assume that Oi = 0.1, 02 = 0.2 
and H = 0.3. Under these conditions, the basin of attraction obtained numerically from 
the theory is displayed in the left panel of Fig.|^. We should note that in the region (b), 
the patterns with activity ai can be recalled, while the patterns with activity 02 cannot. 
Since we consider the case Pi = P2, the vertical lines correspond to half of the usual 
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maximum storage capacities. In the right figures, we display typical behavior of the 
overlap m{t) for the initial condition m(0) = 0.5. The load parameters a = 0.02,0.05 
and 0.08 used here correspond to the regions (a), (b) and (c) shown in the left figure, 
respectively. In the right panel of Fig.^ appear comparisons of the time evolution of 
the overlap for two different values of a in three different regions of a. The evolutions 
corresponding to these values of a represent the retrieval processes of Q and ^f^"*"^ 
associated with the activity levels Oi and 02, respectively. Note that in the region (b), 
the patterns with a = 0.2 act only as noise when the network is recalling the pattern 
with a = 0.1. As a whole, the above findings suggest that the network has a good ability 
to retrieve patterns with different firing rates. We should remark that the patterns with 
activity ai can be stored more stably rather than those with activity a2 when ai < 02. 



4. Dilution 



In this section, we study the influence of random synaptic dilution on the model's 
associative memory capability. For the case of the phase oscillator model, that is, when 



a = 1 and H = 0, this effect has already been reported |TJ, |T^. Following the method 
used in that case to treat random synaptic dilution in our model, we assume that the 
synaptic efficacies take the form 

Cij = —^Cij, (45) 

where Cij is the standard Hebbian matrix, as defined by Eq.(^, and, the Cij are 
independent random variables, taking the values 1 and with probabilities c and 
1 — c, respectively. Note that the dilution parameter c represents the ratio of connected 
synapses. In the limit 00, Eq.(^) can be regarded as 

Cij = Cij + r]ij, (46) 

where the synaptic noise rjij is a complex Gaussian noise with mean and variance 
7f/N The relationship between the dilution parameter c and the variance rj can 
be calculated as 

= i^a. (47) 

c 

Under the same assumption as that used in obtaining Eq.(^), we can separate the local 
field into the signal and two noise parts as follows: 

N 

hm = T.c^,w,{t) 

j 

N 



N T P 

UT]^j:m+vr^)w,{t) 



M'm + Ziit) (48) 
M'(t)^ + zm + ^t(t), (49) 
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where zf{t) and zf{t) are defined as 

1 N p 

m = ^Y.T.mwAt) (50) 

N 

ztit)=Y.rj,,W,{t). (51) 

j 

The new noise term z^{t) is caused by synaptic dilution, while the term z'^{t) is like 
the noise defined by Eg . ([TT|) , representing the crosstalk noise arising from the other 
embedded patterns. In analogy to our treatment in Sec.^, we assume that z^(t) and 
z^{t) are independent complex Gaussian noises with mean and variance o"^(t) and 
c'^it), respectively. Therefore, Zi(t) can be regarded as a complex Gaussian noise with 
mean and variance 2a^{t) = alit) + alit). Thus, we can derive dynamical equations 
in the same way as in Sec.|[ The form of the macroscopic order parameter m{t + 1) is 
the same as that given by Eg. (p^ : 

™(* + i)-^EieJi/(iie]i"W + ^,wi)||=|i||| 

First, applying the theory of statistical neurodynamics to z^{t), we calculate cr^(t). 
However, to apply the theory of statistical neurodynamics, it is necessary to take into 
account the second term in 

4{t + 1) = E%/(l^°l)^ + E^.-^.^ + 4||f) ' ^^^^ 

where /t° = hj{t) —r]jiWi(t). As reported in [jl2|, for the asymmetrical case, rjij ^ rjji, the 
second term here can be ignored. Then, as seen from Fig.(P), there is little difference 
between symmetric and asymmetric cases in a sparse coding system. Thus we conjecture 
that this term can be ignored altogether. Doing so, in the N oo limit, we obtain 

a^M=V^Qim{t),a{t)) 

= ^—^aQ{m{t),a(t)), (54) 
c 

where Q{m(t),(T{t)) is defined by Eg . (|3^) . 

For the crosstalk noise z'^{t), in analogy to Eg. ([TBI) , we obtain 

z'r{t + 1) = K{m{t),a{t)) + zmG{m{t),a{t)), (55) 

where K{m{t),a{t)) and G{m{t),a{t)) are defined by Egs.(|l^) and (plSf ), respectively. 

4.1. Equilibrium State 

In the eguilibrium state, putting z^{t) = z!l{t + 1) into Eg. (p5D , we obtain 
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Therefore, cx^, the variance of z{t), takes the form 



^2 1 ^2 I 1 2 



^2(1-^)2 2c 

Consequently, the properties of the network in the equihbrium state can be calculated 
from Eqs.(^), ([58|), ( ]25|) and (0). In Fig.|10|, we summarize the theoretical results 



concerning the dependence on the ratio of connected synapses c. In Figs.|TO](a) and 
(b), it is found that the maximum storage capacity is an increasing function of the 
connectivity c. Particularly, we can see from (b) that, as the activity level a becomes 
small, the storage capacity decreases almost linearly with the connectivity c. A similar 
linear dependence is observed in the case of the diluted Hopfield model. On the other 
hand, in (c), it is found that even if the ratio of connectivity is small, the maximum 
storage capacity tends to diverge in the limit a ^ 0. However, it seems that the rate of 
this divergence decreases as c decreases. 

4-2. Nan- equilibrium State 

In order to estimate the robustness with respect to synaptic damage, we should also 
consider the influence of synaptic dilution on the retrieval process, particularly, on the 
basin of attraction. For this purpose, we can apply the same method as used in the 
derivation of Eq.(0). After some calculations, the resulting recursion equations at 
second order are given by 

a^t + 1) = |g(m(t), a{t)) + a\t)G\mit), a(t)) 

+ aG{m{t), a{t))X {t + l,t) 

+ aa^mit + l)m{t - l)G(m(t), a{t))G{m{t - 1), a{t - 1)) 
+ 1(1 _ G\m{t), a{t))WQ{m{t), a{t)) (59) 
X{t + l,t) = Re< Wj{t + l)Wj{t) >j (60) 

X{t, t - 1) - G{m{t - 1), a{t - l))Q(m(t - 1), a{t - 1)) ^ 
^ 2a{t)(T{t - 1) ' 

where rj is related to the connectivity c via Eq. (^Tj) . As mentioned in Sec.|^, Eq. ([61|) 
is used to calculate Eq. (|60|) . For initial conditions, we adopt the values used in Sec.|^, 
except in the present case we use cr^(O) = aa/2c. In the case of a = 0.1 and H = 0.5, 



Fig.|TT| illustrates the theoretical results concerning the basins of attraction for various 
values of the connectivity c. We can see that near saturation a ~ ac, the basin of 
attraction remains large even for low connectivity. Therefore, we find that synaptic 
dilution has little influence on the width of the basin of attraction, even though the 
storage capacity decreases with the connectivity. 
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5. Conclusion 

In this paper, we have presented a simple extended model of oscillator neural networks 
to allow for the description of the non-firing state. We have studied the model's 
associative memory capability for sparsely coded phase patterns, in which some neurons 
are in the non-firing state and the other neurons encode information in the phase 
variable representing the timing of neuronal spikes. In particular, applying the theory of 
statistical neurodynamics, we have evaluated the maximum storage capacity and derived 
the basin of attraction. We have found the following properties of our model in its basic 
form: 

• The storage capacity diverges as the activity level decreases to zero. It was 
numerically found that the storage capacity diverges proportionally with 1/alna 
in the limit a — > 0. 

• Even just below the maximum storage capacity, the basin of attraction remains 
large. 

We then investigated the model with regard to the size of the basin of attraction. We 
found that with the model in its basic form, the basin of attraction is smaller than those 
of the binary model and the phase oscillator. For associative memory, it is desirable that 
the basin of attraction be large. For this reason, we considered employing a dynamically 
adjusted threshold, and we found the following: 

• The basin of attraction can be enlarged by using the dynamically adjusted 
threshold. 

In view of biology, the neurons may die of age or be injured by accident. Thus, the 
robustness with respect to synaptic damage is important for real neuronal systems. For 
this reason, we also investigated our model with regard to robustness, and we found the 
following: 

• It was found that the system is robust with respect to synaptic damage: Even in the 
case of a high cutting rate, the basin of attraction remains large, and the maximum 
of the storage capacity diverges in the a — > limit. For low activity patterns, the 
maximum storage capacity decreases almost linearly with the ratio of connected 
synapse. 

The above properties are common with the Hopfield model. In addition, we have found 
that our model possesses a novel feature not seen in the Hopfield model. In realistic 
situations, the activity level of the firing pattern may generally depend on the content 
of the information. Using traditional neural network models based only on firing rate 
coding, however, we encounter difficulties when the network simultaneously stores such 
patterns with different activity levels. Contrastingly, with our model, we found the 
following: 

• Unlike the Hopfield model, provided that the phase distribution in the embedded 
patterns is uniform, it was shown that patterns with different activity levels can be 
memorized simultaneously. 
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In conclusion, from the above findings it is seen tliat the oscillator neural network 
exhibits good performance in the sparse coding situation. Therefore, we believe 
that these results support the plausibility of temporal coding and we hope that 
they encourage attempts for more detailed explorations of the nature of temporal 
coding. As a future work, we wish to consider one of the applications of the 
oscillator model. In this paper we considered coherent oscillations of only a uniform 
distribution. It is more interesting, however, to consider systems that are not so strongly 
constrained. For example, neurons could be organized into ensembles characterized by 
different phase values. In such a situation, each ensemble would carry out a different 
function independently, and the neural system as a whole could carry out several tasks 
simultaneously. Using this kind of synchrony, we may obtain a simple solution to the 
binding problem. 
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Figure 1. (a) The step function f{x) = Q{x — H). where H is the "threshold": When 
\hi{t)\ > H, \W^{t + l)\ = 1, and otherwise, + 1)| = 0. (b) Depiction of the 

dynamical change of the neuronal state. The circle and the origin correspond to the 
firing state and the non- firing state, respectively. The phase and amplitude of Wi{t) 
change according to Eq.(||). 




Figure 2. (a) Storage capacity ac as a function of the mean activity level a for various 
thresholds H. The data points indicate numerical results with N — 1000 for 20 trials. 
The lines were obtained from theoretical analysis, (b) A three-dimensional isometric 
plot of the maximum storage capacity ac as a function of a and H from the theoretical 
results. 
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Figure 3. Time evolution of tlic overlap m(t) for various initial conditions: m = 
0.25,0.31,0.4. Here a = 0.013, H = 0.3 and a = 0.5. Results of the second-order 
approximation give better agreement with the numerical simulation than those of the 
first-order approximation. 
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Figure 4. (a) Basins of attraction for various values of the threshold H, with a = 0.1. 
The data points are the results of the numerical simulation, the dashed lines correspond 
to the first-order approximation, and the solid lines correspond to the second-order 
approximation. The results of the numerical simulation were obtained with = 1000 
for 20 trials, (b) Same as (a) but with a = 0.5. 




Figure 5. (a) Basins of attraction for various firing rates a and thresholds H. The 
data points are the results of the numerical simulations [N = iOOO for 20 trials), and 
the solid lines are the theoretical results, (b) A three-dimensional isometric plot of the 
maximum storage capacity ac as a function of a and H from the theoretical results. 
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Figure 6. Basins of attraction in the case that H{t) is a dynamically adjusted variable 
with a = 0.1. Comparing with Fig.^(a). we find that this mechanism enlarges the 
basin of attraction. The data points represent the results of a numerical simulation 
with N = 1000 for 20 trials, and the line is the theoretical result. 
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Figure 7. Storage capacity aic as a function of a2 for ai = 0.1 and 02 = 0.2. The 
circular data points represent as a function of a2, and the diamond-shaped points 
represent aic + 02. Both of them wore obtained from a numerical simulation with 
N = 2000 for 20 trials. 
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Figure 8. (left) Basin of attraction for patterns with a = 0.1 in the case that patterns 
with a = 0.1 and a = 0.2 are embedded in equal number at the same time. Here 
H = 0.3. The dotted line and the data points represent the theoretical and the 
numerical results, respectively. The vertical long-dashed line indicates the maximum 
storage capacity for patterns with a = 0.2. Thus, in the left figure (a) is the region 
where both activity level patterns are successfully retrieved, (b) is the region where 
only patterns with a = 0.1 can be retrieved, and (c) is the region where neither can 
be retrieved, (right) Time evolutions of the overlap m{t) for the initial condition 
m(0) = 0.5, with H — 0.3 and a = 0.02,0.05 and 0.08, corresponding to the regions 
(a), (b) and (c) in the left figure, respectively. These are the results of the numerical 
simulation. 
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Figure 9. Comparison of basins for symmetrical and asymmetrical dilution. Both 
were obtained from numerical simulations with TV = 2000 for 20 trials. 




Figure 10. (a) Dependence of the maximum storage storage capacity ac on the ratio 
of connectivity c for various firing rates with H ~ 0.5. The solid hnes represent the 
theoretical results, and the data points represent the numerical results with N = 2000 
for 20 trials, (b) Same as (a), but normalized by ac(c)/ac(l). (c) A three-dimensional 
isometric plot of the maximum storage capacity as a function of a and c for H — 0.5. 
This was obtained from the theoretical results. 
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Figure 11. Basins of attraction for various values of the connectivity c. The solid lines 
represent the theoretical results, and the data points represent the numerical results 
with N = 2000 for 20 trials. 



